(19) 



J 



(12) 



EuropSisches Patentamt 
European Patent Office 
Office europ6endes brevets (11) EP 1 033 694 A2 

EUROPEAN PATENT APPLICATION 



(43) 


Date of publication: 


(51) Int. CI. 7 : G08G 1/16 




06.09.2000 Bulletin 2000/36 




(21) 


Application number: 00104079.9 




(22) 


Date of filing: 28.02.2000 




(84) 


Designated Contracting States: 


• Fujinaml, Kazutomo 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


Susono-shi, Shizuoka 410-1107 (JP) 




MCNLPTSE 


• Okamoto, Keikl 




Designated Extension States: 


Hamana-gun, Shizuoka 431-0213 (JP) 




AL LT LV MK RO SI 


(74) Representative: 






(30) 


Priority: 01 .03.1999 JP 5243799 


GrQnecker, Kinkeldey, 


Stockmair & Schwanhdusser 


(71) 


Applicant: Yazaki Corporation 


Anwattssoziet&t 




Minato-ku, Tokyo 108-0073 (JP) 


Maximilianstrasse 58 






80538 MQnchen (DE) 


(72) 


Inventors: 




• 


Ishikawa, Naoto 






Susono-shi, Shizuoka 410-1107 (JP) 





CM 
< 

o> 

CD 

CO 
CO 

o 



(54) Rear-view monitoring system for vehicles 

(57) A rear-view monitor for use in vehicles is pro- 
vided, in which contained are vehicle-mounted image 
pickup means for picking up images of road in the rear 
of one's own vehicle at every fixed time; and detection 
means for detecting an overtaking vehicle by process- 
ing road images obtained by the image pickup means. 
The monitor keeps monitoring of relative movement 
between one's own vehicle and the overtaking vehicle 
detected by the detection means. The monitor is char- 
acterized in that the image pickup means contain a 
wide-angle high resolution camera, and the detection 
means include: first image processing means for 
processing the whole road images obtained by the 
image pickup means by sampling image data; second 
image processing means for processing a part of road 
images obtained by the image pickup means without 
sampling image data; and selection means for selecting 
either the first image processing means or the second 
image processing means in response to a situation of 
traffic. Thus, a rear-view monitor for use in vehicles 
according to the present invention enables monitoring 
over a wide range, i.e. the far and near distances, under 
preferable conditions. 
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Description 

BACKGROUND OF THE INVENTION 

(1) Field of the Invention 

[0001] This invention relates to a rear-view monitor 
for use in vehicles and, more specifically, to a rear-view 
monitor that warns a driver to respond to an vehicle 
approaching from the rear with regard to one's own 
vehicle by taking images of road in the rear of one's own 
vehicle with an image pickup device such as a vedio 
camera mounted on one's own vehicle such as a motor- 
vehicle and then, by detecting an vehicle approaching 
from the rear using road images thus obtained. 

(2) Description of the Related Art 

[0002] Several rear-view monitoring and warning 
systems have hitherto been proposed in the past, for 
example, Japanese Patent Publication No. 2641562, 
Japanese Patent Application LaidOpen No. H6-1 07096 
and H7-50769. The systems are capable of monitoring 
the rear road in order to prevent a possible collision with 
an overtaking vehicle from occuring by correctly sizing a 
situation of the overtaking vehicle traveling an adjacent 
either right or left traffic lane and then, warning a driver 
of possibility of collision with such overtaking vehicle 
traveling the adjacent lane when a driver is about to 
chang the traveling lane. 

[0003] According to the proposed systems, in order 
to detect an overtaking vehicle traveling the adjacent 
lane quickly and securely without performing an exces- 
sive image processing, lanes on the road are distin- 
guished by recognizing white lines, i.e. lane dividing 
lines, using road images taken by a video camera, and 
a monitoring range is set to the adjacent lane in accord- 
ance with this discrimination so that an amount of image 
processing for detecting an approaching vehicle within 
an adjacent lane, i.e. a monitoring range set up in the 
road images taken by a video camera, is reduced. 
[0004] Apparatuses that employ a detection system 
by optical flow are disclosed in Japanese Patent Appli- 
cation Laid-Open No. H6-1 07096 and H7-50769, fea- 
tures of which are expalined with reference to Figs. 7 
and 8 in the following. 

[0005] Rgure 7 is a block diagram illustrating an 
assembly of a conventional rear-view monitoring and 
warning system for vehicles, which contains an image 
pickup unit 10 as an image pickup device such as a 
vedio camera 11 , a data processing unit 30 as a com- 
puting system, and a speaker 42 as a warning means. 
[0006] The image pickup unit 10 is mounted at the 
rear side of a vehicle such as on a trunk lid so as to take 
road images of the rear with regard to the traveling vehi- 
cle. The data processing unit 30 contains a CPU 31 as 
a central processing unit operating in accordance with 
an operation program, a ROM 32 for memorizing the 



operation program of the CPU 31 and predetermined 
settings, and a RAM 33 for provisionally memorizing 
data required upon computation by the CPU 31. The 
speaker 42 mounted on the vehicle warns a driver of a 

5 danger by sounding or warning in accordance with a 
signal from the CPU 31 in the data processing unit 30. 
[0007] Figures 8A and 8B illustrate a change in 
images taken by the image pickup unit 10 mounted at 
the rear portion of the vehicle. Fig. 8A is an image taken 

10 at a time t and Fig. 8B at a time t + At. In the each figure, 
shown are an overtaking vehicle 200 traveling in the 
rear with regard to one's own vehicle, a traffic sign 300 
disposed along a road 500, and a building 400 along the 
road 500. 

is [0008] Supposing that one's own vehicle is traveling 
straight a flat road, with time passing by, i.e. accompa- 
nied by the vehicle traveling, the traffic sign 300 and 
building 400 are relatively leaving from the vehicle, 
resulting in that the images of traffic sign 300 and build- 

20 ing 400 become small. That is, in the figures, images of 
the traffic sign 300 and building 400 taken at the time t 
shown in Fig. 8A is smaller than those taken at the time 
t + A t shown in Fig. 8B. 

[0009] In the following, the optical flows are 
25 expalined with reference to these figures. 

[0010] In these figures, to be considered are a plu- 
rality of marked points selected in the images taken at 
the time t (see Fig. 8A) such as: marked points 201a 
and 202a for the overtaking vehicle 200; 301a, 302a 
30 and 303a for the traffic sign 300; and 401 a and 402a for 
the building 400. Similarly, to be considered are marked 
points such as: 201b and 202b for the overtaking vehi- 
cle 200; 301b, 302b and 303b for the traffic sign 300; 
and 401 b and 402b for the building 400 in the images 
35 taken at the time t + A t (see Fig. 8B). Then, each com- 
bination between corresponding marked points, for 
example 20 1a and 201b, gives velocity vector such as 
201 F, 202F, 301 F, 302F, 303F, 401 F and 402F as shown 
in 8C. These velocity vectors are defined as the optical 

40 flOWS. 

[0011] Here, it can be understood that the optical 
flow radially appears from a focus of expansion (herein- 
after, FOE) defined as an infinitely far point or a vanish- 
ing point in the figures. When one's own vehicle is 

45 traveling straight, the FOE corresponds to just the oppo- 
site direction to which the vehicle is traveling. 
[0012] When one's own vehicle is traveling, an opti- 
cal flow of an object going away from the vehicle is a 
convergent vector heading toward the FOE, while an 

so optical flow of an object approaching to the vehicle is a 
divergent vector away from the FOE. Accordingly, the 
optical flows 201 F and 202F (shown in Fig. 8C) are 
divergent vectors, indicating that the vehicle 200 is 
approaching one's own vehicle, in other wards, that the 

55 vehicle 200 is traveling faster than one's own vehicle. 
[0013] Regarding a size of the optical flow, the size 
is large when a difference in velocity between one's own 
vehicle and an object is large and also when a distance 
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therebetween is short In the following, this matter is 
explained with reference to the attached drawings. 
[001 4] Figure 9 illustrates an optical arrangement of 
the image pickup unit 10, in which 1 la is a lens of the 
video camera in the image pickup unit 10, 11b is an 
image plane of the video camera, f is a distance 
between the lens 11a and the image plane 1 1b, P (X a , 
Y a , ZJ is an arbitrary point of the overtaking vehicle, 
and p (Xa, yj is a point corresponding to the point P on 
the image plane 11b. 

[0015] On this occasion, the following formula is 
given on the basis of similar figures in triangles: 

x a = f • X a /2 a (1) 

Transforming the formula (1) and then, differentiating 
with respect to time gives the following formula (2): 

X a ' = (Ax a /At • Z a + x a • Z a Vf (2) 

A x-component u of an optical flow is given by the follow- 
ing formula (3): 

u = Ax a /At (3) 

Hence, the following formula (4) is derived from the for- 
mula (3): 

Z a = (f • X a '-x a - Z a Vu (4) 

[0016] Here, Z a * denotes a difference in velocity 
between the overtaking vehicle (200 in Fig. 8) traveling 
the same lane or the adjacent lane and one's own vehi- 
cle on which the image pickup unit 10 is mounted. 
Assuming this difference in velocity to be - a , the for- 
mula (4) is expressed by the following formula (5): 

Z a = (f * X a ' + x a • a)/u (5) 

Hence, the x-component u of the optical flow, is 
expressed by the following formula (6): 

u = (f • X a ' + x a • o)/Z a (6) 

By the way, Y a , i.e. Y-coordinate of the point P, can be 
derived in the similar way. 

[0017] Consequently, according to the formula (6), 
when Z is small, i.e. a distance between one's own vehi- 
cle and the overtaking vehicle 200 traveling the same 
lane or the adjacent lane is short, or when a is large, i.e. 
the difference in velocity between one's own vehicle and 
the overtaking vehicle 200 is large, an x-component of 
the optical flow becomes large. These relations are the 
same with respect to the Ydirection. 
[0018] Accordingly, the size of the optical flow 
becomes large when the distance between one's own 
vehicle and the overtaking vehicle 200 is short, and 
when the difference in velocity between the both vehi- 



cles is large, resulting in that the direction of the optical 
flow diverges from the FOE. In this case, the larger the 
size of the optical flow, the larger a degree of danger of 
the collision. 

5 [001 9] The data processing unit 30 recognizes a sit- 
uation, in which the optical flow is the divergent vector 
as mentioned above and also the size of the optical flow 
is large, caused by either or both situations as follows: 
one situation that the object is in the vicinity of one's 

w own vehicle and another situation that the object is 
approaching to one's own vehicle with higher speed 
than that of one's own vehicle, and the data processing 
unit 30 judges that the degree of danger is high. Thus, 
the data processing unit 30 warns a driver of the danger 

is via the speaker when the unit judges that the degree of 
danger is high. 

[0020] By such repeated data processings with 
respect to all the points on the image taken at the time 
t, optical flows covering the whole image can be 

20 obtained, and each degree of clanger corresponding to 
each object is determined. Then, an attention is given to 
a driver by warning sounds and the like according to the 
degree of danger determined, resulting in giving a com- 
plement to a limited recognition by human being and 

25 prevention of an actual traffic accident from occurring or 
being out of dangerous situation which might otherwise 
develop to a serious accident. 

[0021] As shown in Fig. 10, in a conventional art, a 
detection of white lines of a lane, along which one's own 

30 vehicle is traveling a straight six-lane expressway, 
brings about a discrimination between said lane and 
adjacent lanes and a determination of range to be mon- 
itored, aiming for that a processing time for monitoring 
of unnecessary field of vision can be saved and a high- 

35 speed processing can be achieved. An extension of 
thus detected white lines determines a FOE point and 
the overtaking vehicle 200 is detected by determining 
optical flows developed radially from said FOE point 
regarding a range of one's lane and adjacent lanes. 

40 Since a necessary recognition in the system is formed 
on the basis of the size of the optical flows, a degree of 
danger regarding the overtaking vehicle 200 traveling 
the rear or adjacent lanes is automatically judged and 
advantageously, an extra speedometer is not neces- 

45 sary. 

[0022] Among currently proposed detection meth- 
ods of the optical flow, a correlation method is nearly on 
a level of practical use. The correlation method has a 
disadvantage of enormous amount of computation 

so since the method contains a search for corresponding 
points of window (pixel) with respect to alt circumferen- 
tial areas and a computation of correlation values. On 
the other hand, the method has an advantage of finding 
relatively correct optical flow regarding a complicated 

55 image, for which the present invention has as a subject. 
[0023] As mentioned above, according to the corre- 
lation method, when optical flows are to be found 
regarding an image at the time t, searchs in all direc- 
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tions are necessary to find out which pixel among all the 
pixels in an image at the time t corresponds to which 
pixel in an image at the time t- A t, causing enormous 
amount of computation as well as a possible error in 
required response. Hence, it is considered that limiting 
a monitoring range might solve problems such as long 
p recessing time and low detection accuracy. 
[0024] As for the aforementioned type of rear-view 
monitoring system for use in vehicles, from an image 
taken by a video camera, a large image of other vehicle 
in the vicinity of one's own vehicle can be obtained com- 
pared with that far from one's own vehicle, but only a 
small image of other vehicle far from one's own vehicle 
can be obtained. Therefore, an image of low resolution 
is good enough in the vicinity of one's own vehicle, but a 
distant image with low resolution makes it difficult to 
catch a behavior of other vehicle precisely by an image 
processing and consequently, it is preferable that 
images with high resolution as much as possible can be 
obtained regarding distant images. 
[0025] In addition, taking a case of lane change into 
consideration, it is preferable to use a wide-angle video 
camera to monitor adjacent lanes as close as possible 
to one's own vehicle. However, the use of the wide- 
angle video camera makes the monitoring range wider, 
resulting in deterioration in the resolution and increase 
in number of image data to be processed, causing prob- 
lems such as long precessing time and low detection 
accuracy. 

[0026] Thus, regarding this types of monitoring sys- 
tem, in which monitoring of the far and near distances 
are simultaneously needed, there are above-mentioned 
contradictory requirements and therefore, a measure to 
meet with these requirements has been desirable. 

SUMMARY OF THE INVENTION 

[0027] It is therefore an object of the present inven- 
tion to solve the above-mentioned problems and to pro- 
vide a rear-view monitor for use in vehicles, which 
enables monitoring over a wide range, i.e. the far and 
near distances, under preferable conditions. 
[0028] In order to accomplish the above object, 
referring to a basic assembly shown in Fig. 1 A, a first 
aspect of the present invention is to provide a rear-view 
monitor for use in vehicles, comprising: vehicle- 
mounted image pickup means 10 for picking up images 
of road in the rear of one's own vehicle at every fixed 
time; and detection means 31-1 for detecting an over- 
taking vehicle by processing road images obtained by 
the image pickup means, wherein relative movement 
between one's own vehicle and the overtaking vehicle 
detected by the detection means is monitored, the 
image pickup means contain a wide-angle high resolu- 
tion camera, and the detection means include: first 
image processing means 31-11 for processing the 
whole road images obtained by the image pickup 
means by sampling image data; second image process- 



ing means 31-12 for processing a part of road images 
obtained by the image pickup means without sampling 
image data; and selection means 31-13 for selecting 
either the first image processing means or the second 
s image processing means in response to a situation of 
traffic. 

[0029] In the rear-view monitor for use in vehicles 
according to the first aspect of the present invention, the 
detection means 31-1 process the images of road in the 

10 rear of one's own vehicle, of which vehicle-mounted 
image pickup means 10 monitor at every fixed time, to 
detect an overtaking vehicle and relative movement 
between the detected overtaking vehicle and one's own 
vehicle is monitored. The image pickup means contain a 

is wide-angle high resolution camera, and the first image 
processing means 31-11 process the whole road 
images obtained by the image pickup means by sam- 
pling image data, and the second image processing 
means 31-12 process a part of road images obtained by 

20 the image pickup means without sampling image data, 
and the selection means 31-13 select either the first 
image processing means or the second image process- 
ing means in response to a situation of traffic. 
[0030] Therefore, when the selection means select 

25 the first image processing means, the whole road 
images obtained by the image pickup means are proc- 
essed by sampling image data and wide-range images 
are processed with low resolution, enabling a wide- 
range monitoring of other vehicles in the vicinity of one's 

30 own vehicle without spending long processing time. On 
the other hand, when the selection means select the 
second image processing means, the part of road 
images obtained by the image pickup means are proc- 
essed without sampling image data and narrow-range 

35 images are processed with high resolution, enabling a 
monitoring with high resolution of other vehicles being 
far from one's own vehicle without spending long 
processing time. Consequently, provided is a rear-view 
monitor for use in vehicles, which enables monitoring 

40 over a wide range, i.e. the far and near distances, under 
preferable conditions. 

[0031] The second aspect of the present invention 
is to provide the rear- view monitor for use in vehicles 
according to the first aspect of the present invention, 

45 wherein the selection means select: the first image 
processing means when the detection means detects 
other vehicle in the vicinity of one's own vehicle; the 
second image processing means when the detection 
means detects no other vehicle in the vicinity of one's 

so own vehicle; and the first image processing means at a 
low repetition rate at the time when the second image 
processing means being selected. 
[0032] In the rear-view monitor for use in vehicles 
according to the second aspect of the present invention, 

55 when the detection means detect other vehicle in the 
vicinity of one's own vehicle, the selection means select 
the first image processing means so that the monitor 
can monitor the other vehicle in the vicinity of one's own 
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vehicle despite monitoring with low resolution. When 
the detection means detect no other vehicle in the vic- 
ntty of one's own vehicle, the selection means select the 
second image processing means so that the monitor 
can monitor the other vehicle far from one's own vehicle 5 
with high resolution despite monitoring in narrow range. 
Furthermore, when the selection means select the sec- 
ond image processing means, the selection means also 
select the first image processing means at a low repeti- 
tion rate so that the monitor can sometimes monitor the 
other vehicle in the vicinity of one's own vehicle despite 
monitoring with low resolution. Consequently, provided 
is a rear-view monitor for use in vehicles, which enables 
monitoring over a wide range, i.e. the far and near dis- 
tances, under preferable conditions. 
[0033] Referring to a basic assembly shown in Fig. 
1 8, the third aspect of the present invention is to provide 
a rear- view monitor for use in vehicles, comprising: vehi- 
cle-mounted image pickup means 10 for picking up 
images of road in the rear of one's own vehicle at every 
fixed time; and detection means 31-2 for detecting an 
overtaking vehicle by processing road images obtained 
by the image pickup means, wherein relative movement 
between one's own vehicle and the overtaking vehicle 
detected by the detection means is monitored, the mon- 
itor further contains blink detection means 52 for detect- 
ing blinks of winkers and an operated side thereof, the 
image pickup means contain a wide-angle high resolu- 
tion camera, and the detection means include: first 
image processing means 31-21 for processing a left- 
hand side of road images obtained by the image pickup 
means; second image processing means 31-22 for 
processing a right-hand side of road images obtained 
by the image pickup means; and selection means 31-23 
for selecting alternatively the first image processing 
means and the second image processing means and 
maintaining selected image processing means that cor- 
respond to a direction to which the winkers are operated 
when the blink detection means detects the operation of 
the winkers. 

[0034] In the rear-view monitor for use in vehicles 
according to the third aspect of the present invention, 
the first image processing means 31-21 process the 
left-hand side of road images obtained by the image 
pickup means, and the second image processing 
means 31-22 process the right-hand side of road 
images obtained by the image pickup means, and the 
selection means 31-23 select alternatively the first 
image processing means and the second image 
processing means, and the blink detection means 52 
maintain selected image processing means that corre- 
spond to a direction to which the winkers are operated 
when the blink detection means 52 detect the operation 
of the winkers, resulting in that the monitor can monitor 
other vehicle traveling near and far on adjacent lanes 
with high resolution without spending long processing 
time and can always monitor a adjacent lane into which 
one's own vehicle is about to enter when changing 



lanes. Consequently, provided is a rear-view monitor for 
use in vehicles, which enables monitoring of other vehi- 
cles under preferable conditions. 
[0035] Referring to a basic assembly shown in Fig. 
1 C, the fourth aspect of the present invention is to pro- 
vide a rear-view monitor for use in vehicles, comprising: 
vehicle-mounted image pickup means 10 for picking up 
road images in the rear of one's own vehicle at every 
fixed time; and optical flow detection means 31-3 for 
detecting an optical flow of other vehicle determined by 
successive two road images obtained at every fixed 
time by the image pickup means, wherein the monitor 
keeps monitoring relative movement between one's 
own vehicle and an overtaking vehicle using the optical 
flow detected by the optical flow detection means, the 
image pickup means contain a wide-angle high resolu- 
tion camera, and the optical flow detection means 
include: first image processing means 31-31 for 
processing the whole successive two road images 
obtained at every fixed time by the image pickup means 
by sampling image data; second image processing 
means 31-32 for processing a part of road images 
obtained by the image pickup means without sampling 
image data; and selection means 31-33 for selecting 
either the first image processing means or the second 
image processing means in response to a situation of 
traffic. 

[0036] In the rear-view monitor for use in vehicles 
according to the fourth aspect of the present invention, 
the optical flow detection means 31 -3 detect an optical 
flow of other vehicle based on successive two road 
images obtained at every fixed time by the image pickup 
means, and the monitor keeps monitoring of relative 
movement between one's own vehicle and other over- 
taking vehicle using the optical flow detected by the opti- 
cal flow detection means. The first image processing 
means 31-31 process the whole successive two road 
images obtained at every fixed time by the image pickup 
means by sampling image data, and the second image 
processing means 31-32 process a part of road images 
obtained by the image pickup means without sampling 
image data, and the selection means 31 -33 select either 
the first image processing means or the second image 
processing means in response to a situation of traffic, 
resulting in that the monitor can monitor other vehicles 
being in the vicinity of one's own vehicle in wide range 
and can monitor other vehicles being far from one's own 
vehicle with high resolution without spending long 
processing time, by using optical flows of other vehicles, 
even if the image pickup means contain a wide-angle 
high resolution camera. Consequently, provided is a 
rear-view monitor for use in vehicles, which enables 
monitoring of other vehicles under preferable condi- 
tions, without bringing inconvenience accompanied by 
emptying a wide-angle high resolution camera. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[0037] 

Figures 1A, 1B and 1C are basic assemblies of a 
rear-view monitor for use in vehicles according to 
the present invention. 

Figure 2 is a block diagram illustrating an assembly 
of an optical flow-type rear- view monitor for use in 
vehicles according to the present invention. 
Figure 3 illustrates an example of an image plane in 
a video camera. 

Figure 4 is a flow chart illustrating an outline of an 
action in a rear-view monitor for use in vehicles 
according to the present invention. 
Figure 5 illustrates an example of a road image 
taken by a video camera in a rear- view monitor for 
use in vehicles according to the present invention. 
Figure 6 is a flowchart illustrating an example of 
switching process between wide-angle and zooms 
modes. 

Figure 7 is a block diagram illustrating an assembly 
of a conventional rear-view monitoring and warning 
system for vehicles. 

Figures 8A, 8B and 8C illustrate a change in an 
image of the rear road taken by a video camera. 
Figure 9 illustrates an optical arrangement of the 
image pickup unit 10. 

Figure 10 is a conceptual illustration of an image of 
straight six-lane expressway. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

[0038] In the following, the preferred embodiments 
of the present invention are explained with reference to 
the attached drawings. 

[0039] Figure 2 is a block diagram illustrating an 
assembly of an optical flow-type rear-view monitor for 
use in vehicles according to the present invention, in 
which the assembly comprises an image pickup unit 10 
as image pickup means, a memory unit 20 for storing of 
image data and the like from the image pickup unit 10, a 
data processing unit 30 for performing image process- 
ing and monitor processing for relative movement 
between one's own vehicle and other vehicles based on 
image data obtained by the image pickup unit 10, an 
warning unit 40 as warning means giving an warning, 
and a signal input unit 50 for inputting of signal indicat- 
ing operational data upon changing a traveling direction 
of one's own vehicle. 

[0040] The image pickup unit 10 contains a video 
camera 11 that contains a wide-angle lens 11a having 
angle of view of 120 degrees, a high resolution image 
plane 11b having pixels as many as 1024 x 1024, for 
example, as shown in Fig. 3, and a so-called CMOS 
camera, which accesses randomly to any pixel on the 
image plane 11b to read them in. Differently from a 



charge-transfer type, in which charges stored in pixel 
corresponding to an incident light are sequentially sent 
and picked out, the CMOS camera is constructed as a 
X-Y address type so that a pixel storing charges can be 

5 selected by a horizontal line (X) and a vertical line (Y). 
By constructing each scanning circuit, which selects a 
position along corresponding line, so as to be a circuit 
that can randomly accesses using a multi-plexus, a 
pixel can be arbitrarily selected and stored charges 

10 therein can be picked out. 

[0041] The video camera 11 of the image pickup 
unit 10 is mounted on either a trunk lid at the rear side 
of a vehicle or a rear end of a vehicle. The video camera 
1 1 is arranged to take road images in the rear of one's 

15 own vehicle and outputs image data into CPU 31 
(described later) of the data processing unit 30. 
[0042] The memory unit 20 comprises a first frame 
memory 21 storing road image data from the image 
pickup unit 10, the second frame memory 22, and a 

20 divergent optical flow memory 23 storing an optical flow 
in divergent direction. 

[0043] The first frame memory 21 and the second 
frame memory 22 are constructed as a matrix-like 
memory consisting of a matrix m x n, for example, an 

25 512- by 512- pixel matrix 512 x 512. As shown in Frg.3, 
when a wide-angle lens is used, pixel data by sampling 
image data are written by reading from every other pixel 
on the image plane 1 1b, and when a zoom lens is used, 
pixel data corresponding to a part of a whole image are 

30 written by reading from pixels in a pixel matrix 512x512 
in the vicinity of a FOE point. 

[0044] The data processing unit 30 comprises CPU 
31 as a central processing unit operating according to 
an operation program, ROM 32 for storing the operation 

35 program for the CPU 31 and predetermined values, and 
RAM 33 for temporarily storing data that are necessary 
upon earring out an operation by the CPU 31 . 
[0045] The warning unit 40 comprises an indicator 
41 and a speaker 42. The indicator 41 , consisting of a 

40 liquid crystal display (LCD) and the like, displays images 
taken by the video camera 1 1 , and warns a driver of 
danger by an image with indicating a message when the 
data processing unit 30 (the CPU 31) judges a danger 
of contact with other vehicle. The speaker 42 gives a 

45 sound such as sound guidance or warning sound. 
When the data processing unit 30 judges a danger of 
contact with other vehicle existing, the speaker 42 
warns a driver of the danger by the sound. 
[0046] The signal input unit 50 comprises a steering 

so angle sensor 51 as a steering angle detection means for 
detecting a steering angle or an operative steering 
angle for a steering wheel (normally, front wheel) of 
vehicle and a blink (turn signal) detection swith 52 as a 
blink detection means for detecting an operational state 

55 of a winkers mechanism by a driver and a direction 
thereof, in which the steering angle sensor 51 detects a 
turning data of vehicle and the blink detection swith 52 
detects a turning indication data from the winkers mech- 
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anism operated by a driver when the driver turns the 
vehicle to left or right. 

[0047] Now, a control operation by the data 
processing unit 30 is explained with reference to the 
attached flow chart. 5 
[0048] In the control operation, an image taking 
process is performed in a step S110 in a major flow 
chart shown in Fig. 4. The image taking process at the 
step S110 gives a rear road image, for example, as 
shewn in Fig. 5. At an initial stage, the video camera 1 1 
is set up in a wide-angle mode and data from every 
other pixel can be read out. 

[0049] The road image shown in Fig. 5 is an exam- 
ple of image taken from one's own vehicle traveling an 
exclusive road for vehicle such as expressway and is an 
image of the rear road since taken by a video camera 1 1 
that is mounted at the rear of a vehicle. The road image 
includes: a road 500; white lines 510 and 520 drawn on 
the road 500 as broken lines, which section the road to 
one's lane and adjacent lanes and indicates changing 
lane to be possible; white lines 530 and 540 drawn on 
the road 500 as solid lines, which section side roads 
and indicates changing lane to be impossible; and walls 
600 set up at both sides of the road 500. All of the above 
elements in the image vanish at a FOE point located at 
a center horizontally and at one-third position vertically 
in the image. Road image data thus taken are stored in 
the first frame memory 21 of the memory unit 20. 
[0050] Since road images are taken by the video 
camera mounted at the rear of vehicle facing backward, 
the right in the road image corresponds to the left from 
the viewpoint of traveling direction and the left in the 
road image corresponds to the right from the viewpoint 
of traveling direction. Hereinafter, the following descrip- 
tion is done on the basis of the road image regarding 
definition of right and left. 

[0051] Then, an optical flow detection process is 
performed in a step S120 in the flow chart shown in Fig. 
4. Operating according to a predetermined program, the 
CPU 31 functions as the optical flow detection means 
31-3 that detects optical flows arisen from other vehi- 
cles in monitoring range, which is set based on succes- 
sive two road images obtained at every fixed time by the 
video camera 10. In the optical flow detection process, 
an optical flow is detected as a vector formed by con- 
necting two mutually corresponded points, one of which 
is a point in a road image taken at a time t and stored in 
the first frame memory 21, and another of which is the 
corresponding point in a road image taken at a time t + 
A t and stored in the second frame memory 22. Among 
thus detected optical flows, optical flows of divergent 
direction, i.e. divergent optical flows arising from other 
vehicles approaching to one's own vehicle, are stored in 
a divergent optical flow memory 23. 
[0052] A wide-angle / zoom switch process is a next 
step S130 shown in Fig. 4 and is performed according 
to a flow chart shown in Fig. 6. At the beginning, in a 
step S210a, a judge is done whether an optical flow in 



the vicinity of one's own vehicle is stored or not in the 
divergent optical flow memory 23. When an optical flow 
in the vicinity of one's own vehicle is stored, a wide- 
angle mode is set up in a step S220a and then, a coun- 
ter C is cleared in a step S220a\ and the process goes 
back to the beginning. On the other hand, based on the 
judge in a step S210a t when an optical flow in the vicin- 
ity of one's own vehicle is not stored, a zoom mode is 
set up in a step S230a. Then, the counter C increases 
in a step S240a and then, a judge is done whether the 
counter C becomes 10 or not in a step S250a. When the 
counter C becomes 10, a wide-angle mode is set up in 
a step S260a and then, a counter C is cleared in a step 
S270a. and the process goes back to the beginning. 
[0053] According to the above processes, the 
CPU31 of the data processing unit 30 functions as: the 
optical flow detection means 31-3 for detecting an opti- 
cal flow of other vehicle determined by successive two 
road images taken at every fixed time by the video cam- 
era 11 of the image pickup unit 10; the first image 
processing means 31-31 for processing the whole suc- 
cessive two road images taken at every fixed time by the 
video camera 11 of the image pickup unit 10 by sam- 
pling image data; the second image processing means 
31-32 for processing a part of road images taken by the 
video camera 11 of the image pickup unit 10 without 
sampling image data; and the selection means 31-33 
for selecting either the first image processing means or 
the second image processing means in response to a 
situation of traffic. 

[0054] As shown in the flow chart in Fig. 6, when a 
' zoom mode is set up, a wide-angle mode is set up once 
in every ten steps even if an optical flow in the vicinity of 
one's own vehicle is not stored, so that a sudden 
approach of other vehicle to one's own vehicle can be 
checked, while the sudden approach cannot be 
checked in the zoom mode. In this case, successive two 
road images are obtained in one step of wide-angle 
mode to meet with the requirement of optical flow detec- 
tion. 

[0055] A step S140 shown in Fig. 4 is a computing 
process of degree of danger. In the step S140, among 
optical flows obtained in the step S130, a mesurement 
of optical flow size is performed with respect to optical 
flows of divergent direction such as aforementioned 
optical flows 201 F and 202F. In the computing process, 
levels for the degree of danger may be computed by set- 
ting up several threshold levels. 
[0056] In a step S150, based on the computed 
degree of danger in the step S140, a judgment of being 
in danger is done when the degree of danger exceeds a 
threshold value. If levels for the degree of danger are 
given, a judgment of being in danger is done when the 
level for the degree of danger becomes a specified 
value or higher. 

[0057] When a judgment of being in danger is done 
in the step S150, a warning process is performed in a 
following step S160. On the other hand, when a judg- 
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ment of not being in danger is done in the step S150, a 
series of process is finished and a process beginning 
with the step S11 0 is started again as shown in Fig. 4. 
[0058] In the warning process in the step S160, a 
sound signal is sent out to the speaker 42 of the warning s 
unit 40 to make the speaker 42 give a sound such as 
sound guidance or warning sound, thereby drawing a 
driver's attention and warning of a danger to the driver 
with indicating an message and the like through the indi- 
cator 41 . When the warning process in the step S1 60 is 10 
finished, a series of process is finished and a process 
beginning with the step S1 10 is started again. 
[0059] In the aforementioned preferred embodi- 
ment, a wide-angle mode and zoom mode are changed 
alternatively depending upon whether an optical flow is 
exists in the vicinity of one's own vehicle or not. Differ- 
ently from this, regardless of existence or non-existence 
of an optical flow in the vicinity of one's own vehicle, the 
far and near distances may always be monitored by 
constantly changing a wide-angle mode and a zoom 20 
mode alternatively, and a detection of optical flows may 
be performed by using an alternate set of successive 
wide-angle image and zoom image. 
[0060] In addition, such a monitoring may be possi- 
ble that right and left areas alternatively are always 25 
monitored using a zoom mode and an area, into which 
a driver is about to change lanes and enter, is fixed to be 
monitored according to an operation of the blink detec- 
tion switch 52. Also, such a monitoring may be possible 
that a wide-angle mode is always used for monitoring 30 
and an area, into which a driver is about to change 
lanes and enter, is monitored using zoom mode accord- 
ing to an operation of the blink detection switch 52. 
[0061] According to the above processes, the 
CPU31 of the data processing unit 30 functions as: the 35 
detection means 31-2 for detecting other overtaking 
vehicle by processing road images obtained by the 
video camera 11 of the image pickup unit 10; the first 
image processing means 31-21 for processing a left- 
hand side of road images obtained by the image pickup 40 
means; the second image processing means 31-22 for 
processing a right-hand side of road images obtained 
by the video camera 1 1 of the image pickup unit 10; and 
the selection means 31-23 that select alternatively the 
first image processing means and the second image 45 
processing means and maintain selected image 
processing means that correspond to a direction to 
which the winkers are operated when the blink detection 
switch 52 detects the operation of the winkers. 
[0062] In the above-preferred embodiment, shown so 
is an example for using a video camera that can access 
randomly to any pixel on the image plane 1 1b so as to 
deal with without increasing memory capacity. Differ- 
ently from this, when a third frame memory having large 
memory capacity can be prepared besides the first ss 
frame memory 21 and the second frame memory 22, 
images taken by the charge-transfer type video camera 
1 1 having high resolution may tentatively be read into 



the third frame memory and then, data as a pixel unit 
may be randomly accessed from the third frame mem- 
ory so as to proceed a process accordingly. 
[0063] At any rate, in both modes of a wide-angle 
and a zoom, the image processing does not require a 
long time since number of pixels to be processed is not 
large, resulting in that the detection process of optical 
flows can be finished during taking of successive two 
road images, even if a high speed data processing unit 
is not prepared. 

[0064] In the above-preferred embodiment, other 
vehicles are monitored by detecting optical flows 
thereof. Differently from this, according to the present 
invention, any other image processing that does not use 
optical flows can be employed to a system for monitor- 
ing of other vehicles. In such a case, the CPU31 of the 
data processing unit 30, which performs an image 
processing, functions as: the detection means 31-1 for 
detecting other overtaking vehicle by processing road 
images obtained by the video camera 11 of the image 
pickup unit 10; the first image processing means 31-1 1 
for processing the whole road images obtained by the 
video camera 11 of the image pickup unit 10 by sam- 
pling image data; the second image processing means 
31-12 for processing a part of road images obtained by 
the video camera 1 1 of the image pickup unit 10 without 
sampling image data; and the selection means 31-13 
for selecting either the first image processing means or 
the second image processing means in response to a 
situation of traffic. 

Claims 

1 . A rear- view monitor for use in vehicles, comprising: 

vehicle-mounted image pickup means for pick- 
ing up images of road in the rear of one's own 
vehicle at every fixed time; and 
detection means for detecting an overtaking 
vehicle by processing road images obtained by 
the image pickup means, 
wherein relative movement between one's own 
vehicle and the overtaking vehicle detected by 
the detection means is monitored, 
the image pickup means contain a wide-angle 
high resolution camera, and 
the detection means include: 
first image processing means for processing 
the whole road images obtained by the image 
pickup means by sampling image data; 
second image processing means for process- 
ing a part of road images obtained by the 
image pickup means without sampling image 
data; and 

selection means for selecting either the first 
image processing means or the second image 
processing means in response to a situation of 
traffic. 
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2. The rear-view monitor for use in vehicles according 
to daim 1 , wherein the selection means select: 

the first image processing means when the 

detection means detects other vehicle in the s 

vicinity of one's own vehicle; 

the second image processing means when the 

detection means detects no other vehicle in the 

vicinity of one's own vehicle; and 

the first image processing means at a low rep- io 

etition rate at the time when the second image 

processing means being selected. 

3. A rear-view monitor for use in vehicles, comprising : 

15 

vehicle-mounted image pickup means for pick- 
ing up images of road in the rear of one's own 
vehicle at every fixed time; and 
detection means for detecting an overtaking 
vehicle by processing road images obtained by 20 
the image pickup means, 
wherein relative movement between one's own 
vehicle and the overtaking vehicle detected by 
the detection means is monitored, 
the monitor further contains blink detection 25 
means for detecting blinks of winkers and an 
operated side thereof, 

the image pickup means contain a wide-angle 
high resolution camera, and 
the detection means include: 30 
first image processing means for processing a 
left-hand side of road images obtained by the 
image pickup means; 

second image processing means for process- 
ing a right-hand side of road images obtained 35 
by the image pickup means; and 
selection means for selecting alternatively the 
first image processing means and the second 
image processing means and maintaining 
selected image processing means that corre- 40 
spond to a direction to which the winkers are 
operated when the blink detection means 
detects the operation of the winkers. 

4. A rear-view monitor for use in vehicles, comprising : 45 

vehicle-mounted image pickup means for pick- 
ing up images of road in the rear of one's own 
vehicle at every fixed time; and 
optical flow detection means for detecting an so 
optical flow of other vehicle determined by suc- 
cessive two road images obtained at every 
fixed time by the image pickup means, 
wherein the monitor keeps monitoring relative 
movement between one's own vehicle and an 55 
overtaking vehicle using the optical flow 
detected by the optical flow detection means, 
the image pickup means contain a wide-angle 



high resolution camera, and 
the optical flow detection means include: 
first image processing means for processing 
the whole successive two road images 
obtained at every fixed time by the image 
pickup means by sampling image data; 
second image processing means for process- 
ing a part of road images obtained by the 
image pickup means without sampling image 
data; and 

selection means for selecting either the first 
image processing means or the second image 
processing means in response to a situation of 
traffic. 
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